Papers with initialization strategies
Evaluating Lottery Tickets Under Distributional Shifts (D19-61)
Copied to clipboard
| Challenge: | Recent research suggests deep neural networks are dramatically over-parametrized. |
| Approach: | They propose that large, over-parameterized neural networks consist of small, sparse subnetworks that can be trained in isolation to reach a similar (or better) test accuracy. |
| Outcome: | The proposed models can achieve commensurate performance using the same initialization as the original model. |
EnerGIZAr: Leveraging GIZA++ for Effective Tokenizer Initialization (2025.findings-acl)
Copied to clipboard
| Challenge: | Continual pre-training has long been considered the default strategy for adapting models to non-English languages, but struggles with initializing new embeddings, especially for non-Latin scripts. |
| Approach: | They propose a method that leverages statistical word alignment techniques to improve continual pre-training by leveraging word alignment matrix between source and target tokens. |
| Outcome: | The proposed method outperforms existing methods on key NLP tasks including POS tagging, Sentiment Analysis, NLI, and NER in Hindi, Basque, Arabic and Korean. |
Embedding Semantic Taxonomies (2020.coling-main)
Copied to clipboard
| Challenge: | Recent work on hierarchical representational structures in machine learning promises to blend the value of human curated taxonomies with the power and flexibility of machine learning systems. |
| Approach: | They propose to use box embeddings to encode aspects of partial ordering property of taxonomies to represent a medical subject headings taxonomy. |
| Outcome: | The proposed model outperforms baselines for taxonomic reconstruction and bipartite relationship experiments and is compared with a set of 300K PubMed articles with subject labels from MeSH. |